fix(ci): staleness gate fails on named defects, not on the calendar#522
Merged
Conversation
MEASURED 2026-07-21: 294 of 350 consumers were red on the single pin d7c2271 (2026-06-26) — every one of them green a fortnight earlier, with nothing changed in any consumer. aspasia is the clean natural experiment: same pin, 2026-07-09 run at 32 commits / 12d = passing notice; 2026-07-21 at 63 commits / 24d = hard error. standards moves ~2.6 commits/day, so a consumer exhausts both window budgets ~14 days after any propagation. Holding the fleet green under that rule means re-pinning ~300 repos every fortnight (order 7,800 PRs/year). A gate that fails everyone on a timer is not a guard: it trains the estate to ignore it, and it buries the findings that matter — the same sweep shows 148 real Workflow security linter failures and 74 anti-pattern failures that read as noise once the fleet is uniformly red. 1. Age outside the window is now a ::notice, not an error. The window is still computed and still reported, so propagate-workflow-pins.sh and the Hypatia sha_bump_propagation rule keep their signal. 2. A named deny-list (KNOWN_BAD_BEFORE) replaces age as the hard failure. Each entry is <reusable>:<fix-sha>; a pin that is a strict ancestor of the fix carries that defect and is rejected at any age. This is what the window was only ever a proxy for, and it is strictly better in both directions: a RECENT pin carrying the defect is now caught, and an old pin carrying none is no longer punished by the calendar. First entry is e9c8888 (#441). Before it, hypatia-scan-reusable and the validate-hypatia-baseline job cached the built Hypatia scanner under a keyless key while the build steps were guarded by `if [ ! -d ]`, so the first scanner build ever cached was reused forever and scanner fixes never took effect. A pin older than this reports a FALSE GREEN. All 294 consumers on d7c2271 predate it by one day and stay red — now for a true and actionable reason. 3. Integrity no longer rests on the runner's clone. The UNKNOWN / "may be forged" verdict was firing on legitimate pins: awesome-haskell pins governance-reusable@5a93d9d5 and was accused on four consecutive runs over 17 days, while that commit verifies as a true ancestor of main locally in both treeless and --depth 200 clones and via the compare API (behind=0, ahead=90). The mechanism was never reproduced off-runner. A hard FORGED verdict now requires either a COMPLETE local clone (not shallow, not partial) or confirmation from GET /repos/{nwo}/compare/{pin}...{branch}. Where neither is available the gate warns and passes: "cannot verify" is not "compromised". Zero API calls on the happy path — the server is consulted only when about to accuse. The deny-list gets the same fallback, so a degraded runner cannot silently skip it; "could not check" is reported as SKIPPED, never as passed. Tests: the hermetic fixture suite goes 12 -> 16 cases. Out-of-window now asserts exit 0 AND the advisory notice (a new run_case_out helper — an exit code alone cannot distinguish "passed silently" from "passed with the notice the propagation path depends on"). New coverage: deny-list rejects at any age, rejects even when in-window, does not reject the fixing commit itself, and is scoped per reusable. 16/16 pass, and the suite stays hermetic because a complete fixture clone resolves the forged case without the network. Also verified against live standards history and in degraded-clone mode; both give identical verdicts for fresh / denied / old-but-clean / forged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
hyperpolymath
marked this pull request as ready for review
July 21, 2026 15:25
Owner
Author
Follow-up commit: two holes closed, found by testing the path production actually takesCases 1–12 of the fixture suite all run against a complete 1. The deny-list's negative still trusted local A treeless clone resolves every commit, so the API fallback was gated on "commits do not resolve" and was dead code in CI. Trace That is the exact false green this list exists to prevent, on the very cohort that proved the bug is real. Now a local positive is still trusted (no false positives observed, and a deny must not need the network), but a local negative is accepted only from a complete clone — otherwise it is confirmed server-side. 2.
3. The server-side confirmation is unauthenticated — 60/hr per runner IP — and exhausting it degrades the gate to Tests: 16 → 19, still hermeticThe API is pointed at a closed port (
|
hyperpolymath
added a commit
that referenced
this pull request
Jul 21, 2026
…low probe (#524) Follow-up to #522, which merged before this landed. **#522 is live on `main` now, so these two defects are live on ~300 consumers.** Both were found by adding a test for the path production actually takes. Cases 1–12 of the fixture suite run against a **complete** `git init` repo; CI runs against a **partial** clone (`--filter=tree:0`). Nothing tested the code that protects production, and it had two bugs. ## 1. The deny-list's *negative* still trusted local `merge-base` on a partial clone A treeless clone resolves every commit, so the API fallback was gated on "commits do not resolve" — making it **dead code in CI**. Trace `awesome-haskell` (pin `5a93d9d5`, which predates the Hypatia fix and *must* be denied): ``` pin_is_known_bad -> local branch taken -> wrong negative -> continue -> NOT denied classify_pin -> same mis-fire -> API says ahead=90 -> OUT_OF_WINDOW -> notice -> PASS, while running the frozen scanner ``` That is the exact false green the deny-list exists to prevent, on the very cohort that proved `merge-base` is unreliable there. **Fix:** a local **positive** is still trusted (no false positives observed, and a deny must not require the network), but a local **negative** is accepted only from a *complete* clone — otherwise it is confirmed server-side. ## 2. `clone_is_complete` used a cwd-relative git-dir `rev-parse --git-dir` returns `.git`, and the check was `[ -e "$gd/shallow" ]` — which resolves against the **CWD**, i.e. the *consumer's* checkout, which `actions/checkout` makes shallow by default. So `.git/shallow` would nearly always exist and **every** standards clone would be classified incomplete, permanently masking the complete-clone fast path. Now uses `--absolute-git-dir`. Test 13c failed for exactly this reason, which is how it was found. ## 3. `governance-reusable.yml` passes `GITHUB_TOKEN` to the staleness step The server-side confirmation is unauthenticated — 60/hr per runner IP — and exhausting it degrades the gate to `::warning`. That fail-open is deliberate and **loud** (two warnings naming the pin; never a silent pass), and the common path never calls the API at all, since a genuine deny is a local positive. A token removes the cliff. This reaches a consumer only once it re-pins past this commit — the script is read from `standards` HEAD, but the job definition comes from the pinned SHA — so it lands naturally with the propagation the deny-listed pins need anyway. ## Tests: 16 → 19, still hermetic The API is pointed at a closed port (`http://127.0.0.1:1`), so these are fast and offline: ``` PASS: partial clone: local positive still denies without the network PASS: partial clone: unverifiable deny-list check is reported SKIPPED, not passed PASS: complete clone: local negative is trusted silently (no SKIPPED warning) ``` `run_case_out` gained a leading-`!` inversion so a test can assert the gate did **not** say something. Suite verified from two different working directories, since defect (2) was cwd-dependent. 19/19 pass on top of current `main`; `shellcheck -S warning` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Governance staleness gate is currently failing 294 of 350 consumer repos — and it is failing them for the wrong reason.
What is actually happening
All 294 pin the same SHA,
d7c22711(2026-06-26). Nothing is wrong with any of them individually.standardsmoves ~2.6 commits/day, so a consumer exhausts both window budgets (≤50 commits or ≤14 days) about a fortnight after any propagation, and goes red on a timer.aspasiais the clean natural experiment — same pin, nothing changed in the repo:::notice, passing::error, hard failHolding the fleet green under that rule means re-pinning ~300 repos every fortnight — on the order of 7,800 PRs/year.
A gate that fails everyone on a timer is not a guard. It trains the estate to ignore it, and it buries the findings that matter: the same sweep shows 148 real
Workflow security linterfailures and 74 anti-pattern failures, all of which read as noise once the fleet is uniformly red.What changes
1. Age outside the window is now a
::notice, not an error. The window is still computed and still reported, sopropagate-workflow-pins.shand the Hypatiasha_bump_propagationrule keep their signal.2. A named deny-list (
KNOWN_BAD_BEFORE) replaces age as the hard failure. Each entry is<reusable>:<fix-sha>; a pin that is a strict ancestor of the fix carries that defect and is rejected at any age.This is what the window was only ever a proxy for, and it is strictly better in both directions — a recent pin carrying the defect is now caught, and an old pin carrying none is no longer punished by the calendar.
The first entry is
e9c8888769a7(#441). Before it,hypatia-scan-reusableand thevalidate-hypatia-baselinejob cached the built Hypatia scanner under a keyless key while the build steps were guarded byif [ ! -d ]— so the first scanner build ever cached was reused forever and scanner fixes never took effect. A pin older than this reports a false green.3. Integrity no longer rests on the runner's clone. The
UNKNOWN/ "may be forged" verdict was firing on legitimate pins.awesome-haskellpinsgovernance-reusable@5a93d9d5and was accused on four consecutive runs over 17 days, while that commit verifies as a true ancestor ofmainlocally in both treeless and--depth 200clones and via the compare API (behind=0, ahead=90). The mechanism was never reproduced off-runner.A supply-chain accusation must not depend on the health of a clone the gate does not control. A hard
FORGEDverdict now requires either a complete local clone (not shallow, not partial) or confirmation fromGET /repos/{nwo}/compare/{pin}...{branch}. Where neither is available the gate emits a::warningand passes — cannot verify is not compromised. Cost is zero API calls on the happy path; the server is consulted only when the gate is about to accuse.The deny-list gets the same fallback, so a degraded runner cannot silently skip it: "could not check" is reported as SKIPPED, never as passed.
Verification
The hermetic fixture suite goes 12 → 16 cases, 16/16 passing:
Out-of-window now asserts exit 0 and the advisory notice, via a new
run_case_outhelper — an exit code alone cannot distinguish "passed silently" from "passed with the notice the propagation path depends on".The suite stays hermetic: a complete fixture clone resolves the forged case locally, so no network is required.
Also verified against live standards history and in degraded-clone mode; both give identical verdicts for fresh / denied / old-but-clean / forged.
shellcheck -S warningclean.🤖 Generated with Claude Code